Note to self: I need to address this after 1.0 is out, xref: cupy/cupy#9801

Q: What would be our recommended way of using InMemoryProgramCache in a multi-GPU env? Wondering about this because we usually have each GPU driven by a thread, and if the intended use case is a global cache object (which makes sense on a homogeneous system like DGX) this would cause serialization.

In CuPy internally there is a per-device cache so this issue is avoided.

Should we merge this section with CUDA compilation toolchain above, or at least move it and make them stay closer?

Alternatively, just merging with the above section is fine too.

This is no-op only for trivial backends. If users don't use it as a context manager (well... it's a separate discussion why they should..), then they better call .close() for consistency and portability (to another backend).

ObjectCode has many public constructors. We must not teach about any private methods.

Why don't we require each cache= instance to subclass from ProgramCacheResource?

cpdef functions are also accessible from Python, is this change needed?

Consideration: logs silently receives nothing on cache hit.

When the cache hits, no compilation occurs, so logs.write() is never called. A caller relying on logs to confirm compilation ran (or to capture PTX output) will get silence with no indication it was a cache hit vs. a compile that produced no output. Worth a note in the cache= or logs parameter docstring, e.g.:

On a cache hit, no compilation is performed and logs receives no output.

Consideration: _decide_driver() is called 2–3 times per key derivation for PTX inputs.

validate() calls self._decide_driver() at line 494, option_fingerprint() calls it again at line 515, and hash_version_probe() calls _linker_backend_and_version() which re-probes independently. Each call re-imports and re-invokes _decide_nvjitlink_or_driver().

If the result were to differ between calls within the same invocation (transient import failure recovering), the key would be internally inconsistent — validation used one backend, fingerprint used another. Even though this is unlikely in practice, computing the result once and threading it through (or caching it on the instance for the duration of one make_program_cache_key call) would be cleaner and more robust.

Consideration: target_type is not case-normalized, asymmetric with code_type.

code_type is lowered here to ensure "PTX" and "ptx" route the same way. But target_type is not normalized — make_program_cache_key(target_type="PTX") and make_program_cache_key(target_type="ptx") produce different keys for the same compilation. The comment on lines 728–730 says "a caller that passes 'PTX' or 'C++' must get the same routing" but only addresses code_type.

Program.compile may normalize target_type internally, but standalone callers of make_program_cache_key are exposed to this instability. Consider adding target_type = target_type.lower() here to match.

Consideration: _LINKER_RELEVANT_FIELDS and _LINKER_FIELD_GATES must be kept in sync manually.

A field added to _LINKER_FIELD_GATES but forgotten in _LINKER_RELEVANT_FIELDS would be silently ignored — the gate exists but is never iterated over in _linker_option_fingerprint. The reverse (missing from _LINKER_FIELD_GATES) would KeyError at runtime, which is fail-fast but late.

A test assertion like assert set(_LINKER_RELEVANT_FIELDS) == set(_LINKER_FIELD_GATES.keys()) would close both directions cheaply. Same applies for sync with _translate_program_options in _program.pyx — the existing test_make_program_cache_key_supported_targets_matches_program_compile guards target drift but not field drift.

Nitpick: __len__ double-stats every entry.

_iter_entry_paths already filters with entry.is_file() (line 531), so every yielded path is known to be a file. The path.is_file() check here in __len__ is redundant — it issues a second stat syscall per entry. For a large cache this is 2N unnecessary stat calls. Safe to remove the inner check:

def __len__(self) -> int:
    return sum(1 for _ in self._iter_entry_paths())

Nitpick: max_size_bytes=0 is accepted but makes the cache a black hole.

Both InMemoryProgramCache and FileStreamProgramCache validate max_size_bytes >= 0, so max_size_bytes=0 passes. But a zero-byte cap means every write is immediately evicted (any payload has size > 0). This is almost certainly a user error. Consider either rejecting it with a ValueError, or documenting it explicitly as "effectively disables caching."

Nitpick: ptxas_options fingerprint preserves list order.

["-v", "-O2"] and ["-O2", "-v"] produce different cache keys even if ptxas treats them equivalently. This causes spurious cache misses (not collisions), so it's safe but suboptimal. The conservative choice (preserve order) is defensible if ptxas flag order is significant for some flags. Worth a brief comment noting the choice is intentional.

Nitpick: Top-level import creates a fragile circular-import dependency.

_keys.py imports ProgramOptions from cuda.core._program at module level, while _program.pyx does a deferred import of make_program_cache_key inside compile(). The deferred import breaks the cycle today, but if someone later adds from cuda.core.utils import ... at the top of _program.pyx, it would break. Worth a # NOTE: ... comment here documenting the mutual dependency and why _keys.py gets the top-level import while _program.pyx defers.

Nitpick: options.name is double-hashed for the linker (PTX) path.

For PTX inputs, "name" is in _LINKER_RELEVANT_FIELDS and goes through _linker_option_fingerprint via _gate_identity. Then this universal block hashes it again under the "options_name" label. Not a collision or instability bug (hashing the same value twice under different labels is safe), but it's redundant work for the linker path. Worth a comment noting the intentional redundancy, or dedup by skipping the linker path here.

Consideration: Temp file accounting under burst writes can cause committed-entry thrashing.

_enforce_size_cap counts temp files toward total but can only evict committed entries (the entries list). During high-concurrency writes, many young temp files (each under the 1h stale-sweep threshold) could push total persistently above the cap with no way to recover — the loop would keep evicting committed entries while the in-flight temps hold the total above max_size_bytes. Once the burst subsides and the temps are committed/cleaned, the evicted entries are gone.

This is a soft-cap design so it's consistent with the documented contract, but worth noting that burst write concurrency can cause disproportionate eviction of committed entries.

Consideration: _enforce_size_cap is O(n) on every __setitem__.

Every write stats all files in entries/ plus tmp/ to compute the total. For a large cache (thousands of entries), this could be measurably costly on every compile. An incremental size tracker (add on write, subtract on eviction, periodic reconciliation to correct drift from external deletions) would make writes O(1) in the common case, falling back to a full scan only when reconciliation is needed.

Consideration: UTF-8 decode introduces a new failure mode in the cache path.

If the source code contains non-UTF-8 bytes (e.g. Latin-1 encoded comments), this .decode("utf-8") raises UnicodeDecodeError even though the uncached compile path would succeed (NVRTC accepts raw bytes). The cache path introduces a failure that doesn't exist without cache=.

This is likely rare in practice (CUDA source is almost always ASCII/UTF-8), but worth documenting in the cache= parameter docstring or catching with a clear error message pointing to the limitation.

Nitpick: Stat-key triple (st_ino, st_size, st_mtime_ns) is repeated in 4 places.

This exact pattern appears in _touch_atime (twice — fd-based and path-based), _prune_if_stat_unchanged, and _enforce_size_cap. Extracting a small helper would reduce duplication and make the invariant easier to audit:

def _stat_key(st: os.stat_result) -> tuple:
    return (st.st_ino, st.st_size, st.st_mtime_ns)

Then each site becomes if _stat_key(st_now) != _stat_key(st_before): return.

Nitpick: Windows sharing-violation retry pattern is repeated across 3 functions.

_replace_with_sharing_retry, _stat_and_read_with_sharing_retry, and _unlink_with_sharing_retry all iterate over _REPLACE_RETRY_DELAYS, sleep, try an operation, catch PermissionError, and check _is_windows_sharing_violation. A generic retry helper parameterized on the operation could deduplicate this:

def _with_sharing_retry(op, *args, on_exhausted=None, **kwargs):
    last_exc = None
    for delay in _REPLACE_RETRY_DELAYS:
        if delay:
            time.sleep(delay)
        try:
            return op(*args, **kwargs)
        except PermissionError as exc:
            if not _is_windows_sharing_violation(exc):
                raise
            last_exc = exc
    if on_exhausted is not None:
        return on_exhausted(last_exc)
    raise last_exc

Not critical — the current code is clear and correct — but it's 3× the same loop structure.

Nitpick: _KeyBackend class hierarchy vs plain function dispatch.

Three stateless singleton classes (_NvrtcBackend, _LinkerBackend, _NvvmBackend) for what is essentially a 3-way dispatch on code_type. The ABC does document the contract clearly (6 abstract/overridable methods), but three plain functions (or a dict of named tuples of functions) dispatched from make_program_cache_key would be simpler and equally extensible — there will likely never be a 4th code_type. The class hierarchy adds indirection and cognitive overhead (readers must trace through the ABC to understand the dispatch) without a clear extensibility payoff.

Not a correctness issue — the current design works — just noting the complexity cost relative to a simpler alternative.